Initialising ...
Initialising ...
Initialising ...
Initialising ...
Initialising ...
Initialising ...
Initialising ...
Hasegawa, Yuta; Onodera, Naoyuki; Asahi, Yuichi; Idomura, Yasuhiro
Dai-36-Kai Suchi Ryutai Rikigaku Shimpojiumu Koen Rombunshu (Internet), 5 Pages, 2022/12
This study implemented and tested the ensemble data assimilation (DA) of turbulent flows using the lattice Boltzmann method and the local ensemble transform Kalman filter (LBM-LETKF). The computational code was implemented fully on GPUs. The test was carried out for the 3D turbulent flow around a square cylinder with 
meshes and 32 ensemble members using 32 GPUs. The time interval of the DA in the test was a half of the period of the Kalman vortex shedding. The normalized mean absolute errors (NMAE) of the lift coefficient were 132%, 148%, and 13.2% for the non-DA case, the nudging case (a simpler DA algorithm), and the LETKF case, respectively. It was found that the LETKF achieved good DA accuracy even though the observation was not frequent enough for the small scale turbulence, while the nudging showed systematic delays in its solution, and could not keep the DA accurately.
Hasegawa, Yuta; Aoki, Takayuki*; Kobayashi, Hiromichi*; Idomura, Yasuhiro; Onodera, Naoyuki
Keisan Kogaku Koenkai Rombunshu (CD-ROM), 26, 6 Pages, 2021/05
We introduce an improved domain partitioning method called "tree cutting approach" for the aerodynamics simulation code based on the lattice Boltzmann method (LBM) with the forest-of-octrees-based local mesh refinement (LMR). The conventional domain partitioning algorithm based on the space-filling curve (SFC), which is widely used in LMR, caused a costly halo data communication which became a bottleneck of our aerodynamics simulation on the GPU-based supercomputers. Our tree cutting approach adopts a hybrid domain partitioning with the coarse structured block decomposition and the SFC partitioning in each block. This hybrid approach improved the locality and the topology of the partitioned sub-domains and reduced the amount of the halo communication to one-third of the original SFC approach. The code achieved 
speedup on 8 GPUs, and achieved 
speedup at the performance of 2207 MLUPS (mega-lattice update per second) on 128 GPUs with strong scaling test.
Hasegawa, Yuta; Onodera, Naoyuki; Idomura, Yasuhiro
Keisan Kogaku Koenkai Rombunshu (CD-ROM), 25, 4 Pages, 2020/06
We developed a GPU-based CFD code using a mesh-refined lattice Boltzmann method (LBM), which enables ensemble simulations for wind and plume dispersion in urban cities. The code is tuned for Pascal or Volta GPU architectures, and is able to perform real-time wind simulations with several kilometers square region and several meters of grid resolution. We examined the developed code against the field experiment JU2003 in Oklahoma City. In the comparison, wind conditions showed good agreements, and the ensemble-averaged and maximum values of tracer concentration satisfied the factor 2 agreements.
Aoki, Takayuki*; Hasegawa, Yuta
Jidosha Gijutsu, 74(4), p.18 - 23, 2020/04
Aerodynamics studies for bicycle racings have been carried out by using a CFD simulation based on LES model. For running of alone cyclist and 2-4 cyclists groups, the computational drags are in good agreement with the wind-tunnel experiments. Different shapes of group running and competing two teams are studied. A large-scale computation for a group of 72 cyclists has been performed by using 2.23 billion meshes on a GPU supercomputer.
Onodera, Naoyuki; Idomura, Yasuhiro
Proceedings of 26th International Conference on Nuclear Engineering (ICONE-26) (Internet), 7 Pages, 2018/07
A large-scale simulation of the environmental dynamics of radioactive substances is very important from the viewpoint of nuclear security. Recently, GPU has been emerging as one of high performance devices to realize a large-scale simulation with less power consumption. We design a plume dispersion simulation based on the AMR-based LBM. We measure the performance of the LBM code on the GPU-rich supercomputer TSUBAME 3.0 at Tokyo Tech. We achieved good weak scaling from 4 GPUs to 144 GPUs, and 30 times higher node performance with CPUs. The code is validated against a wind tunnel test which was released from the National Institute of Advanced Industrial Science and Technology (AIST). The computational grids are subdivided by the AMR method, and the total number of grid points is reduced to less than 10% compared to the finest meshes. In spite of the fewer grid points, the turbulent statistics and plume dispersion are in good agreement with the experiment data.
Onodera, Naoyuki
no journal, ,
The SPEEDI and its world version (WSPEEDI) were developed to predict the off-site diffusion behavior of radioactive substances covering wide areas at ~100km scale based on a mesoscale metrological model. In this work, we apply two new ingredients, GPUs and an adaptive mesh refinement (AMR) method to the lattice Boltzmann method (LBM). In this report, we confirmed the good scalability on the GPU-rich supercomputer, and our code can reproduce the wind tunnel experiment. We conclude that the present LBM is one of most promising approaches to realize a real-time simulation.
Onodera, Naoyuki; Idomura, Yasuhiro; Ali, Y.*; Shimokawabe, Takashi*
no journal, ,
We have developed a communication reduced multi-time-step (CRMT) algorithm for the Post-K supercomputer, and measured the performance on the GPU-based supercomputers. This algorithm is based on the temporal blocking method, and can improve computational efficiency by replacing a communication bottleneck with additional computation. The proposed method is easily applied to the explicit time integration scheme, and is implemented on an extreme scale airflow simulation code CityLBM. We evaluate the performance of the CRMT algorithm on GPU based supercomputers, TSUBAME and Reedbush. Thanks to the CRMT algorithm, the communication cost is reduced by 64%, and weak and strong scaling are improved up to 200 GPUs. The obtained performance indicates that real time airflow simulations for about 2 km square area with the wind speed of 5m/s is feasible using 1m resolution. We conclude that the CRMT algorithm is indispensable for the AMR-LBM to realize a real time simulation on future exascale systems.
Onodera, Naoyuki
no journal, ,
The simulation for dissipation of radioactive substances attract high social interest, and it is required to satisfy both the rapidity and the accuracy. To perform a real-time simulation with high resolution mesh for the scale of human living area such as alleyways and buildings, it is required to develop simulation schemes which can fully utilize high computational performance. In this study, we introduced a nudging-based data assimilation method into the lattice Boltzmann method (LBM), so that we can performe plume dissipation simulations for urban area.
Onodera, Naoyuki; Idomura, Yasuhiro; Kawamura, Takuma; Nakayama, Hiromasa; Shimokawabe, Takashi*; Aoki, Takayuki*
no journal, ,
The simulation for dispersion of radioactive substances attract high social interest, and it is required to satisfy both the speed and the accuracy. To perform a real-time simulation with high resolution mesh for the scale of human living area involving alleyways and buildings, it is required to develop simulation schemes which can fully utilize high computational performance. In this study, we introduced a nudging-based data assimilation method and a plant canopy model into the lattice Boltzmann method (LBM), and confirmed the accuracy of plume dispersion simulations for urban areas is improved.
Hasegawa, Yuta
no journal, ,
To realize the large-scale LES simulation for the aerodynamics of complex shape bodies and the local wind analysis of urban areas, multiple GPU computation of the lattice Boltzmann method (LBM) with adaptive mesh refinement has been implemented. In this presentation, we will explain optimization techniques for the developed code such as single GPU optimization, an optimization of MPI communication, and a spacial parallel implementation for intra-node multiple GPU computation on the latest GPU platforms.
Hasegawa, Yuta; Onodera, Naoyuki; Idomura, Yasuhiro
no journal, ,
A wind simulation code "CityLBM" has been developed for the prediction of plume dispersion in urban areas. CityLBM utilizes a locally mesh-refined lattice Boltzmann method, where turbulent winds near the ground is captured by refining the mesh on low altitude region, and be able to perform real-time wind simulations with a meter-resolution using GPU-accelerated systems. In this study, we validated an ensemble simulation against the field experiment in Oklahoma City (JU2003) with the 100 ensemble members, computational region of 4096 m 
4096 m (horizontal) 2560 m (vertical) and the mesh resolution of 4 m near the ground and buildings. In the comparison between the ensemble simulation and the field experiment, wind speed showed good agreements within the range of 1
(where is variance of ensemble simulation), and the average value of the tracer gas concentration satisfied the factor2 agreements.
Onodera, Naoyuki; Hasegawa, Yuta; Asahi, Yuichi; Idomura, Yasuhiro; Shimokawabe, Takashi*; Aoki, Takayuki*
no journal, ,
A real-time simulation of the environmental dynamics of radioactive substances is very important from the viewpoint of nuclear security. We develop a tracer dispersion simulation code named CityLBM based on a Lattice Boltzmann Method (LBM) with a block-based Adaptive Mesh Refinement (AMR) method. We have currently achieved a real time simulation over a 4 km square area with 2-m resolution using 36 V100 GPUs. In this presentation, we will optimize this code for the latest Ampere architecture on Wisteria/BDEC-01, and show its execution performance.
Hasegawa, Yuta; Idomura, Yasuhiro; Onodera, Naoyuki; Asahi, Yuichi
no journal, ,
The authors are developing a lattice Boltzmann method-local ensemble transformed Kalman filter (LBM-LETKF) to enable the ensemble data assimilation for turbulence with GPUs. The state vector (simulation variables) and observation vector (quantities that can be measured from experiments) have a significant impact on the performance of LETKF: as the state vector, the na
ve method uses a 27-elements vector composed of the LBM velocity distribution functions. However, it is also possible to use the 4-elements vector of macroscopic quantities composed of density and velocity. In this study, we compare the calculation accuracy and speed of the above two methods and select a state vector suitable for turbulence data assimilation.